1. Field of the Invention
The invention concerns image processing, and more particularly temporal interpolation of images working from parent images, with movement compensation.
Interpolation of an image sequence working from another sequence comprises generating an image of the final sequence from images of the original sequence by spatial and temporal interpolation. The purpose of this interpolation is to determine the characteristics (luminance value and if applicable chrominance value) of each pixel of the image to be generated, from the characteristics (luminance value and if applicable chrominance value) of the corresponding pixels in the original images surrounding it. In the following the terms frame and image will be used interchangeably, and only the cases where the input and output frames are described by the same number of lines and points per line will be considered. The other cases can be processed in the same way with the addition of appropriate spatial filtering which will not be described here.
Numerous applications of image interpolation exist. It is used notably as the standard method of converting television image sequences from a 50 Hz standard to a 60 Hz standard or vice versa. It can also be used in coding, for sequence restitution, when frame selection has been performed in the coder. It also more generally enables the image frequency in a sequence to be increased, for example to improve visual comfort.
2. Description of the Prior Art
For image interpolation, a first technique comprises interpolating using a linear space-time filter, this filter being fixed for all the points in the frame and being a function only of the durations separating the input frames from the frames to be generated. It is known that these methods are suitable for the areas or zones of the image which are fixed in time or moving very little, but have considerable defects, in particular for zones with rapid movement. These defects stem from the difficulty in finding a satisfactory compromise between the limitation of the temporal pass band (which creates haziness) and the elimination of the undesirable spectral components (which creates beats).
To avoid these defects, the interpolating filter must be appropriate for the spectrum of the input signal, and this is what is done by a so-called compensated movement interpolation. This technique comprises interpolating the pixels in the direction of movement vectors which are associated with them. To do so the interpolation stage is preceded by a movement estimation operation, which attributes a movement vector to each pixel to be interpolated, this vector in fact giving the address of the same pixel in the input frames which surround the frame to be interpolated. This technique clearly requires the problem of movement analysis to be satisfactorily solved, and in particular, to obtain good image quality, requires the field of movement to be precisely defined, a movement vector being estimated for each pixel and with a precision of better than a pixel. Suitable movement estimators are known, which are capable of reliably supplying a precise movement vector field. Such a movement estimator is, for example, described in French patent application no. 88 12468, entitled "Process and device for movement estimation in a sequence of moving images".
Two types of movement estimators for image interpolation can be distinguished, depending whether the movement is estimated for the pixels in the frame to be interpolated, or for the pixels in one of the parent frames. If the latter, this estimation must be followed by definition of the field of movement of the frames to be interpolated. A method of interpolation using a technique of this type is described in French patent application no. 87 07814, entitled "Process for temporal image interpolation and device for implementing this process". This method can be used notably when several intermediate frames have to be interpolated from two parent frames: a single movement estimation is made between the two parent frames, then the field of movement of each frame to be interpolated is determined from this estimated field. Later we shall consider the case where the field of movement has been obtained for the frame to be interpolated, whatever the method used to achieve this result.
The interpolation itself thus takes these temporal changes into account. In fact the input and output sequences of the interpolator are two representations of the same continuous and generally dynamic scene, at different series of instants. The contents of the sequence are spatial and temporal in nature and the transfer of information from one sequence to another necessarily takes these temporal changes into account. Thus, for compensated movement image interpolation, the pixel (picture element) is no longer considered as a simple picture element but rather as an object element, able to change in the image during time, appear, move, change luminance, and/or disappear. It is clearly essential to adapt to the movement of the pixels, this movement being a very common cause of change in time; taking it into account means that most of the situations possible in a scene can be processed efficiently. Nevertheless, situations can be easily imagined where the movement analysis is incorrect because the movement as a model of temporal change is not appropriate, for example when objects appear or disappear from one input image to the next.
In fact, to take all possible situations into account, it would be necessary to consider, in addition to the movement of the objects, other causes of temporal variation of luminance in time, such as the intrinsic variation in the luminance of object elements (without consideration of movement), and the possible appearance or disappearance of pixels as object elements. By analogy with what has just been said, the simplest method of interpolation, linear interpolation, assumes that the object elements are fixed and considers that the variation in luminance of the pixels with time is due only to a variation in luminance of the object elements, without movement of these elements.
In a device for compensated movement temporal image interpolation, defects can appear in the interpolation when the movement as a model of temporal change is not appropriate, as for complex dynamic scenes, for example when two slender objects cross, or when the movement of an object makes another object disappear from one image to the next, or when a movement is combined with a variation in luminance.
Exact identification of such changes in time would require sophisticated processing out of proportion to the movement estimation necessary for interpolation, to the frequency of occurrence of this type of situation, and to the sensitivity of the eye to this type of event. However it is impossible to use the information supplied by the movement estimator just as it is, because it is far too noisy and generates extremely inconvenient defects.